In [17]:
__author__ = 'Brian Merino <brian.merino@noirlab.edu>, Vinicius Placco <vinicius.placco@noirlab.edu>'
__version__ = '20240606' # yyyymmdd; version datestamp of this notebook
__keywords__ = ['gnirs','gemini','GRB','dragons']

Gemini GNIRS gamma ray burst reduction using DRAGONS Python API¶

Public archival data from gnirsimg_tutorial - GN-2011B-Q-34 (GRB120116A)¶

adapted from https://dragons.readthedocs.io/projects/gnirsimg-drtutorial/en/v3.1.0/01_overview.html¶

Table of contents¶

  • Goals
  • Summary
  • Disclaimers and attribution
  • Imports and setup
  • About the dataset
  • Downloading data for reduction
  • Set up the DRAGONS logger
  • Create File Lists
  • Create Master Dark
  • Create Master Flat Field
  • Reduce Science Images
  • Display stacked final image
  • Clean-up (optional)

Goals¶

Showcase how to perform GNIRS keyhole imaging data reduction using the Gemini DRAGONS package on the Data Lab science platform. Uses a custom DRAGONS kernel "DRAGONS (Py3.7)". The steps include downloading data from the Gemini archive, setting up a DRAGONS calibration service, processing of flats, darks, and science frames, and finally the creation of a single combined stacked image.

Summary¶

DRAGONS is a Python-based astronomical data reduction platform written by the Gemini Science User Support Department. It can currently be used to reduce imaging data from Gemini instruments GMOS, NIRI, Flamingos 2, GSAOI, and GNIRS, as well as spectroscopic data in GMOS longslit mode. Linked here is a general list of guides, manuals, and tutorials about the use of DRAGONS.

The DRAGONS kernel has been made available in the Data Lab environment, allowing users to access the routines without being dependent on installing the software on their local machines.

In this notebook, we present an example of a DRAGONS Jupyter notebook that works in the Data Lab environment to reduce example Gemini North GNIRS J-band imaging data fully. This notebook will not present all of the details of the many options available to adjust or optimize the DRAGONS GNIRS data reduction process; rather, it will just show one example of a standard reduction of a GNIRS imaging dataset.

The data used in this notebook example is GNIRS J-band keyhole imaging from the Gemini archive of the gamma-ray burst GRB120116A from the Gemini North program "Investigating gamma-ray bursts and their use as cosmological probes", PI: Nial Tanvir, program ID GN-2011B-Q-34.

Disclaimer & attribution¶

Disclaimers¶

Note that using the Astro Data Lab constitutes your agreement with our minimal Disclaimers.

Acknowledgments¶

If you use Astro Data Lab in your published research, please include the text in your paper's Acknowledgments section:

This research uses services or data provided by the Astro Data Lab, which is part of the Community Science and Data Center (CSDC) Program of NSF NOIRLab. NOIRLab is operated by the Association of Universities for Research in Astronomy (AURA), Inc. under a cooperative agreement with the U.S. National Science Foundation.

If you use SPARCL jointly with the Astro Data Lab platform (via JupyterLab, command-line, or web interface) in your published research, please include this text below in your paper's Acknowledgments section:

This research uses services or data provided by the SPectra Analysis and Retrievable Catalog Lab (SPARCL) and the Astro Data Lab, which are both part of the Community Science and Data Center (CSDC) Program of NSF NOIRLab. NOIRLab is operated by the Association of Universities for Research in Astronomy (AURA), Inc. under a cooperative agreement with the U.S. National Science Foundation.

In either case please cite the following papers:

  • Data Lab concept paper: Fitzpatrick et al., "The NOAO Data Laboratory: a conceptual overview", SPIE, 9149, 2014, https://doi.org/10.1117/12.2057445

  • Astro Data Lab overview: Nikutta et al., "Data Lab - A Community Science Platform", Astronomy and Computing, 33, 2020, https://doi.org/10.1016/j.ascom.2020.100411

If you are referring to the Data Lab JupyterLab / Jupyter Notebooks, cite:

  • Juneau et al., "Jupyter-Enabled Astrophysical Analysis Using Data-Proximate Computing Platforms", CiSE, 23, 15, 2021, https://doi.org/10.1109/MCSE.2021.3057097

If publishing in a AAS journal, also add the keyword: \facility{Astro Data Lab}

And if you are using SPARCL, please also add \software{SPARCL} and cite:

  • Juneau et al., "SPARCL: SPectra Analysis and Retrievable Catalog Lab", Conference Proceedings for ADASS XXXIII, 2024

https://doi.org/10.48550/arXiv.2401.05576

The NOIRLab Library maintains lists of proper acknowledgments to use when publishing papers using the Lab's facilities, data, or services.

For this notebook specifically, please acknowledge:

  • DRAGONS publication: Labrie et al., "DRAGONS - Data Reduction for Astronomy from Gemini Observatory North and South", ASPC, 523, 321L

  • DRAGONS open source software publication

Importing Python libraries¶

In [2]:
import glob

from recipe_system import cal_service
from recipe_system.reduction.coreReduce import Reduce
from recipe_system.utils.reduce_utils import normalize_ucals

from gempy.adlibrary import dataselect
from gempy.utils import logutils

from astropy.io import fits
from astropy.wcs import WCS

import matplotlib.pyplot as plt
from matplotlib.colors import Normalize

About the dataset¶

This is a GNIRS acquisition keyhole imaging observation of a point source. The observation sequence uses dither-on-target. Dithered observations nearby in time will be used for sky subtraction of each frame.

The calibrations we use for this example include:

  • Darks for the science frames
  • Flats, as a sequence of lamps-on and lamps-off exposures
Observation Type File name(s) Purpose and Exposure (seconds)
Science N20120117S0014-33 J-band, on-target
Science darks N20120102S0538-547 60 s, like Science
Flats N20120117S0034-41 lamps-on
Flats N20120117S0042-49 lamps-off

A note about finding the darks in the GOA: GNIRS is not an imager and imaging through the keyhole is done only in extreme circumstances, so the archive does not have calibration association rules for the darks. One needs to search for the darks manually. Here is the search that was done to find the darks for this observation sequence:

  • Set a date range around the dates of the science observations. In this case, we used "20120101-20120131".
  • Set Instrument to GNIRS.
  • Set Obs. Type to DARK.
  • Set the exposure time to 60 seconds.
  • On the result table, select the darks with a "Pass" setting in the "QA" column.

Downloading the data¶

Downloading J-band images from the Gemini archive to the current working directory. This step only needs to be executed once.

If you run this notebook for the first time and need to download the dataset, set the variable "download=True". The notebook will not redownload the dataset if it is set to False. This will become particularly useful if you run the notebooks more than once.

In [3]:
%%bash 

# create file that lists FITS files to be downloaded
echo "\
http://archive.gemini.edu/file/N20120117S0014.fits
http://archive.gemini.edu/file/N20120117S0015.fits
http://archive.gemini.edu/file/N20120117S0016.fits
http://archive.gemini.edu/file/N20120117S0017.fits
http://archive.gemini.edu/file/N20120117S0018.fits
http://archive.gemini.edu/file/N20120117S0019.fits
http://archive.gemini.edu/file/N20120117S0020.fits
http://archive.gemini.edu/file/N20120117S0021.fits
http://archive.gemini.edu/file/N20120117S0022.fits
http://archive.gemini.edu/file/N20120117S0023.fits
http://archive.gemini.edu/file/N20120117S0024.fits
http://archive.gemini.edu/file/N20120117S0025.fits
http://archive.gemini.edu/file/N20120117S0026.fits
http://archive.gemini.edu/file/N20120117S0027.fits
http://archive.gemini.edu/file/N20120117S0028.fits
http://archive.gemini.edu/file/N20120117S0029.fits
http://archive.gemini.edu/file/N20120117S0030.fits
http://archive.gemini.edu/file/N20120117S0031.fits
http://archive.gemini.edu/file/N20120117S0032.fits
http://archive.gemini.edu/file/N20120117S0033.fits
http://archive.gemini.edu/file/N20120102S0538.fits
http://archive.gemini.edu/file/N20120102S0539.fits
http://archive.gemini.edu/file/N20120102S0540.fits
http://archive.gemini.edu/file/N20120102S0541.fits
http://archive.gemini.edu/file/N20120102S0542.fits
http://archive.gemini.edu/file/N20120102S0543.fits
http://archive.gemini.edu/file/N20120102S0544.fits
http://archive.gemini.edu/file/N20120102S0545.fits
http://archive.gemini.edu/file/N20120102S0546.fits
http://archive.gemini.edu/file/N20120102S0547.fits
http://archive.gemini.edu/file/N20120117S0034.fits
http://archive.gemini.edu/file/N20120117S0035.fits
http://archive.gemini.edu/file/N20120117S0036.fits
http://archive.gemini.edu/file/N20120117S0037.fits
http://archive.gemini.edu/file/N20120117S0038.fits
http://archive.gemini.edu/file/N20120117S0039.fits
http://archive.gemini.edu/file/N20120117S0040.fits
http://archive.gemini.edu/file/N20120117S0041.fits
http://archive.gemini.edu/file/N20120117S0042.fits
http://archive.gemini.edu/file/N20120117S0043.fits
http://archive.gemini.edu/file/N20120117S0044.fits
http://archive.gemini.edu/file/N20120117S0045.fits
http://archive.gemini.edu/file/N20120117S0046.fits
http://archive.gemini.edu/file/N20120117S0047.fits
http://archive.gemini.edu/file/N20120117S0048.fits
http://archive.gemini.edu/file/N20120117S0049.fits\
" > gnirs.list
In [4]:
%%bash

download="True"

if [ $download == "True" ]; then
    wget --no-check-certificate -N -q -i gnirs.list

else
    echo "Skipping download. To download the data set used in this notebook, set download=True."
fi
In [5]:
# # Check header of one raw science image
# tmp = fits.open("N20120117S0014.fits")
# tmp[0].header

Setting up the DRAGONS logger¶

DRAGONS comes with a local calibration manager that uses the same calibration association rules as the Gemini Observatory Archive. This allows reduce to make requests to a local light-weight database for matching processed calibrations when needed to reduce a dataset.

This tells the system where to put the calibration database. This database will keep track of the processed calibrations we will send to it.

In [6]:
logutils.config(file_name='gnirs_data_reduction.log')
caldb = cal_service.set_local_database()
caldb.init("w")

Create a list of all the FITS files in the directory

In [7]:
all_files = glob.glob('N2012*[0-9].fits')
all_files.sort()

Create file lists¶

This data set contains science and calibration frames. Some programs could have different observed targets and exposure times depending on how you organize your raw data.

The DRAGONS data reduction pipeline does not organize the data for you. You have to do it. DRAGONS provides tools to help you with that.

The first step is to create lists that will be used in the data reduction process. For that, we use dataselect. Please refer to the dataselect documentation for details regarding its usage.

In [8]:
darks60 = dataselect.select_data(
    all_files,
    ['DARK'],
    [],
    dataselect.expr_parser('exposure_time==60')
)

flats = dataselect.select_data(all_files, ['FLAT'])

target = dataselect.select_data(
    all_files,
    [],
    [],
    dataselect.expr_parser('object=="GRB120116A"')
)

Create master dark¶

We first create the master dark for the science target. The name of the output master dark, N20120102S0538_dark.fits, is written to the screen at the end of the process.

In [9]:
reduce_darks = Reduce()
reduce_darks.files.extend(darks60)
reduce_darks.runr()

All submitted files appear valid:
N20120102S0538.fits ... N20120102S0547.fits, 10 files submitted.

Ginga not installed, use other viewer, or no viewer

Create master flat field¶

A GNIRS master flat is created from a series of lamp-on and lamp-off exposures. Each flavor is stacked, and then the lamp-off stack is subtracted from the lamp-on stack.

In [10]:
reduce_flats = Reduce()
reduce_flats.files.extend(flats)
reduce_flats.runr()

Reduce science images¶

The science target is a point source. The sequence dithers on-target, moving the source across the thin keyhole aperture. The sky frames for each science image will be the adjacent dithered frames obtained within a certain time limit. The default for GNIRS keyhole images is "within 600 seconds".

Both the master dark and the master flat are in our local calibration database. The calibration manager would retrieve any other Gemini facility instrument automatically. However, GNIRS is not an imager, and the keyhole is typically used only for acquisition, so there are no calibration association rules between GNIRS keyhole images and darks.

The output stack units are in electrons (header keyword BUNIT=electrons). The output stack is stored in a multi-extension FITS (MEF) file. The science signal is in the "SCI" extension, the variance is in the "VAR" extension, and the data quality plane (mask) is in the "DQ" extension.

NOTE: After running this cell, you may receive a WARNING about too few correlated objects. This will not affect the final product, so you can ignore it.

In [11]:
reduce_target = Reduce()
reduce_target.files.extend(target)
reduce_target.uparms = [('darkCorrect:dark', 'N20120102S0538_dark.fits')]
reduce_target.runr()

   WARNING - Too few correlated objects (1). Setting rotate=False, scale=False

Display the stacked image¶

In [12]:
image_file = "N20120117S0014_image.fits"
hdu_list = fits.open(image_file)
wcs = WCS(hdu_list[1].header)
hdu_list.info()

Filename: N20120117S0014_image.fits
No.    Name      Ver    Type      Cards   Dimensions   Format
  0  PRIMARY       1 PrimaryHDU     206   ()      
  1  SCI           1 ImageHDU        50   (1287, 1047)   float32   
  2  VAR           1 ImageHDU        50   (1287, 1047)   float32   
  3  DQ            1 ImageHDU        52   (1287, 1047)   int16 (rescales to uint16)   
  4  PROVENANCE    1 BinTableHDU     17   22R x 4C   [28A, 128A, 128A, 128A]   
  5  PROVHISTORY    1 BinTableHDU     17   17R x 4C   [128A, 428A, 28A, 28A]
In [13]:
image_data = fits.getdata(image_file, ext=1)
print(image_data.shape)

(1047, 1287)
In [14]:
plt.figure(figsize = (10,10))
plt.subplot(projection=wcs)
plt.imshow(image_data,cmap='bone',norm=Normalize(vmin=1, vmax=500),origin='lower')
plt.xlabel('Right Ascension [hh:mm:ss]',fontsize=14,fontweight='bold')
plt.ylabel('Declination [degree]',fontsize=14,fontweight='bold')
plt.show()

Optional: remove raw data (uncomment lines before running)¶

In [16]:
# %%bash

# cp N20120117S0014_image.fits final.fits
# rm -r calibrations
# rm gnirs.list gnirs_data_reduction.log N2012*
# mv final.fits N20120117S0014_image.fits
In [ ]: